Skip to content

fix(gateway): hold a request while its engine's lease drains - #340

Merged
T-Crypt merged 2 commits into
mainfrom
T-Crypt/hold-requests-while-draining
Nov 15, 2026
Merged

T-Crypt merged 2 commits into
mainfrom
T-Crypt/hold-requests-while-draining

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

On aphotic at 12:45:20 on 2026-11-15, the drain read Strata idle between two agent turns and began to unload it. Strata took about 4 seconds to stop and kept answering meanwhile. The next turn came through the gateway at 12:45:22, and Strata ended it with 0 tokens and finish=length. The client got a reply that looked like success and had no answer in it.

The gateway decides an engine is live when its probe answers, and a stopping engine still answers. It already skips a parked engine for the same reason.

Change

  • fit.Leaving reports when an engine's lease is draining, parking or releasing.
  • resolve no longer counts that engine as live, so the request takes the start path, which streams keep-alive comments while it waits.
  • ensure reads the lease every 250 ms until it settles. If the lease is active again (a drain the holder refused), the request goes on to the engine. If the lease is gone, the engine stops answering, and the gateway starts it anew, waiting for the card like any start.

The same wait covers two other paths. A switch to an engine that is still draining now starts it once the drain ends. Before, the switch saw "already up" and did nothing. A request for a ComfyUI that is parking now resumes it once it is parked.

Requests sent straight to an engine's own port bypass the gateway and are not covered. Strata answering a request with 0 tokens while it stops is a Strata bug, filed upstream as Niko1221/Strata#6055.

Proof

  • TestGatewayHoldsARequestWhileItsEngineDrains fails at the test commit (answered while the engine drained: 200). With the fix, it holds the request while the lease reads draining, then forwards it once to the engine when the lease reads active.
  • The full proof passes: build, gofmt, vet, go test ./..., the Windows build and vet, and golangci-lint 2.14.0.
  • Not run against a live drain on aphotic, because that would take the card from the operator's session.

A draining engine still answers its probe, so the gateway proxied the next
request to it. On aphotic at 12:45:22 on 2026-11-15, the runner read Strata
idle, began to unload it, and Strata took the next agent turn in the two
seconds it spent stopping and ended it with 0 tokens (finish=length).

resolve no longer treats an engine as live while its lease is draining,
parking or releasing, and ensure waits for that lease to settle before it
probes. Active again, the request goes on to the engine. Gone, the engine
stops answering and the gateway starts it anew, waiting for the card as any
start does.
@T-Crypt
T-Crypt merged commit cf19258 into main Nov 15, 2026
2 checks passed
@T-Crypt
T-Crypt deleted the T-Crypt/hold-requests-while-draining branch November 15, 2026 21:24
T-Crypt added a commit that referenced this pull request Nov 15, 2026
…ning

Merge main into next: hold requests while an engine drains (#340)
@T-Crypt T-Crypt mentioned this pull request Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant